Image capturing apparatus, image capturing method, and image processing apparatus

ABSTRACT

An image capturing apparatus includes an image data acquirer configured to acquire pieces of image data at a plurality of relative movement positions to which an imaging surface of an imaging sensor is relatively moved with respect to an object light flux; an exposure changer configured to change an exposure when acquiring the pieces of image data at the plurality of relative movement positions; a dynamic range adjuster configured to adjust a dynamic range of the acquired pieces of image data; and an image data combiner configured to obtain composite image data of the pieces of image data, based on a positional shift amount between the pieces of image data and the dynamic range.

CROSS-REFERENCE TO RELATED APPLICATION

The present application is based on and claims priority under 35 U.S.C.§ 119 to Japanese Patent Application No. 2018-054060, filed on Mar. 22,2018, and Japanese Patent Application No. 2019-021125, filed on Feb. 8,2019, the contents of which are incorporated herein by reference intheir entirety.

BACKGROUND OF THE INVENTION 1. Field of the Invention

The present invention relates to an image capturing apparatus, an imagecapturing method, and an image processing apparatus.

2. Description of the Related Art

Patent Document 1 discloses an image capturing apparatus that relativelyshifts an optical input means and an image capturing means for opticallyinputting an object image. Based on a plurality of pieces of color imageinformation of the same object obtained by the shifting, color imageinformation corresponding to one screen is obtained with a certaincolor, and then, based on the color image information of the certaincolor, color image information for one screen is obtained. Accordingly,it is possible to prevent the occurrence of pseudo color and camerashake, and to obtain high definition (high image quality, high accuracy)images.

Patent Document 2 discloses an imaging apparatus including a firstimaging mode of forming video signals by repeating light accumulationfor n seconds, and a second imaging mode of alternately performing lightaccumulation of n seconds and light accumulation of m (<n) seconds, andcombining the n second accumulated images and the m second accumulatedimages, to form video signals of one screen. By the n-second accumulatedimages in the regular first imaging mode, control is implemented to formswitching signals for switching between the first and second imagingmodes, and n-second accumulated images in the second imaging mode andpixel composite control signals of the n-second accumulated images,thereby substantially enlarging the dynamic range.

Patent Document 1: Japanese Unexamined Patent Application PublicationNo. H10-336686

Patent Document 2: Japanese Unexamined Patent Application PublicationNo. H01-060156

SUMMARY OF THE INVENTION

An aspect of the present invention provides an image capturingapparatus, an image capturing method, and an image processing apparatusin which one or more of the disadvantages of the related art arereduced.

According to one aspect of the present invention, there is provided animage capturing apparatus including an image data acquirer configured toacquire pieces of image data at a plurality of relative movementpositions to which an imaging surface of an imaging sensor is relativelymoved with respect to an object light flux; an exposure changerconfigured to change an exposure when acquiring the pieces of image dataat the plurality of relative movement positions; a dynamic rangeadjuster configured to adjust a dynamic range of the acquired pieces ofimage data; and an image data combiner configured to obtain compositeimage data of the pieces of image data, based on a positional shiftamount between the pieces of image data and the dynamic range.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a schematic configuration of anelectronic apparatus with a camera unit mounted with an image capturingapparatus, an image capturing method, an image capturing program, animage processing apparatus, an image processing method, and an imageprocessing program according to a first embodiment of the presentinvention;

FIGS. 2A to 2D are conceptual diagrams illustrating an example of amulti-shot composite mode according to the first embodiment of thepresent invention;

FIGS. 3A to 3D are conceptual diagrams illustrating an example of fourpieces of image data whose exposure is changed at each relative movementposition according to the first embodiment of the present invention;

FIG. 4 is a functional block diagram illustrating an example of aninternal configuration of an image processing apparatus (processor)according to the first embodiment of the present invention;

FIGS. 5A and 5B are conceptual diagrams illustrating an example ofdynamic range adjustment processing by a dynamic range adjusting unitaccording to the first embodiment of the present invention;

FIGS. 6A and 6B are conceptual diagrams illustrating an example of imagedata combining processing by an image data combining unit according tothe first embodiment of the present invention;

FIGS. 7A to 7D are conceptual diagrams illustrating an example ofcombining processing of respective pieces of image data after dynamicrange adjustment according to the first embodiment of the presentinvention;

FIGS. 8A to 8D are first conceptual diagrams illustrating an example ofinterpolation processing of a defective pixel by a pixel interpolatingunit according to the first embodiment of the present invention;

FIG. 9 is a second conceptual diagram illustrating an example ofinterpolation processing of a defective pixel by the pixel interpolatingunit according to the first embodiment of the present invention;

FIGS. 10A to 10D are third conceptual diagrams illustrating an exampleof interpolation processing of a defective pixel by the pixelinterpolating unit according to the first embodiment of the presentinvention;

FIG. 11 is a fourth conceptual diagram illustrating an example ofinterpolation processing of a defective pixel by the pixel interpolatingunit according to the first embodiment of the present invention;

FIG. 12 is a first flowchart illustrating an example of image capturingprocessing according to the first embodiment of the present invention;

FIG. 13 is a second flowchart illustrating an example of image capturingprocessing according to the first embodiment of the present invention;

FIG. 14 is a third flowchart illustrating an example of image capturingprocessing according to the first embodiment of the present invention;

FIG. 15 is a functional block diagram illustrating the internalconfiguration of an image processing apparatus (processor) according toa second embodiment of the present invention;

FIGS. 16A to 16D are conceptual diagrams illustrating an example ofdividing each of a plurality of images into corresponding image areasaccording to the second embodiment of the present invention;

FIG. 17 is a flowchart illustrating an example of image capturingprocessing according to the second embodiment of the present invention;

FIGS. 18A and 18B are a rear view and a cross-sectional view,respectively, of an example configuration of a vibration-proof unitaccording to a third embodiment of the present invention;

FIG. 19 is a rear view of a movable stage of the vibration-proof unitaccording to the third embodiment of the present invention;

FIG. 20 is an enlarged cross-sectional view of an X drive unit includingan X-direction magnet and an X-drive coil according to the thirdembodiment of the present invention;

FIG. 21 is an enlarged cross-sectional view of a Z drive unit includinga Z-direction magnet, a Z-drive coil, and a Z-direction Hall elementaccording to the third embodiment of the present invention;

FIGS. 22A and 22B are diagrams illustrating adverse effects of imageblur in the rotational direction within an XY plane; and

FIG. 23 is a diagram illustrating an example in which a plurality ofimages is divided into image areas having different sizes according tothe third embodiment of the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

In the imaging apparatuses of Patent Documents 1 and 2, the imagecapturing time, the image capturing processes, and the capacity of thememory and the central processing unit (CPU) are increased, and theconfiguration is complicated, and the image quality has not been veryhigh.

A problem to be addressed by an embodiment of the present invention isto provide an image capturing apparatus, an image capturing method, andan image processing apparatus that are simple in structure and that canachieve excellent image quality.

Embodiments of the present invention will be described by referring tothe accompanying drawings.

First Embodiment

FIG. 1 is a block diagram illustrating a schematic configuration of anelectronic apparatus 1 equipped with a camera unit in which an imagecapturing apparatus, an image capturing method, an image capturingprogram, an image processing apparatus, an image processing method, andan image processing program according to the first embodiment areinstalled. The image capturing method, the image capturing program, theimage processing method, and the image processing program according tothe first embodiment are implemented by causing a computer built in theelectronic apparatus 1 to execute predetermined processing steps.

Examples of the electronic apparatus 1 include various kinds ofapparatuses equipped with an image-capturing capability such as adigital camera, a mobile phone, and a game machine. In the firstembodiment, examples in which the electronic apparatus 1 is a digitalcamera are described. Alternatively, the electronic apparatus 1 may bevarious other apparatuses such as personal computers (PCs) that receivean image and perform image processing on the image, without animage-capturing capability.

The digital camera as the electronic apparatus 1 includes, inside acamera body CB, a camera unit (image data acquiring means) 10, an imageprocessing apparatus (processor) 20, a memory (for example, a randomaccess memory (RAM)) 30, a recording medium (for example, UniversalSerial Bus (USB) memory) 40, a display device (for example, a liquidcrystal display (LCD)) 50, an input device 60, a sensor 70, avibration-proof unit 80, a central processing unit (CPU) 90, and anexposure changing unit 100, which are directly or indirectly connectedto each other via a bus 110. The exposure changing unit 100 may be in amode of being installed as one of the elements of the CPU 90 and/or maybe in a mode of being installed as an element separate from the CPU 90(for example, in a mode of being built in the camera unit 10). Note thatthe image processing apparatus (processor) 20 and the CPU 90 may beconfigured by the same hardware device or may be configured as separatehardware devices.

The camera unit 10 has an imaging optical system (not illustrated) andan image sensor (imaging element) 11 (FIG. 2). The imaging opticalsystem forms an image of an object (object image) on the light-receivingsurface of the image sensor 11, and the image sensor 11 converts theformed image into electrical signals using a plurality of pixels havingdifferent detection colors arranged in a matrix. The electrical signalsare then transmitted to the image processing apparatus 20 as an image.The image processing apparatus 20 performs predetermined imageprocessing on the image captured by the camera unit 10. The imageprocessed by the image processing apparatus 20 is temporarily recordedin the memory 30. The image recorded in the memory 30 is stored in therecording medium 40 according to the selection and determination by theuser and displayed on the display device 50.

The input device 60 includes, for example, a power switch, a releaseswitch, a dial for selecting and setting various functions, a four-wayswitch, and a touch panel. The sensor 70 includes, for example, anacceleration sensor, an angular velocity sensor, and an angularacceleration sensor for detecting the acceleration, the angularvelocity, and the angular acceleration of the body of the digital camera(the electronic apparatus 1), respectively. The output of the sensor 70is transmitted to the CPU 90 as a shake detection signal indicatingshaking of the body of the digital camera (the electronic apparatus 1).

The vibration-proof unit 80 moves at least one of the imaging opticalsystem and the image sensor 11 of the camera unit 10, as a moving member(drive member), in a direction different from the direction of theoptical axis of the imaging optical system (for example, within a planeorthogonal to the optical axis of the imaging optical system). The CPU90 controls driving of the vibration-proof unit 80. The CPU 90 receivesa shake detection signal indicating shaking of the body of the digitalcamera from the sensor 70 and causes the vibration-proof unit 80 to movethe moving member in a direction different from the direction of theoptical axis of the imaging optical system. With such a configuration,the image-forming position of the object image is shifted on the imagesensor 11 so that the image blurring due to camera shake can becorrected. The configuration of the vibration-proof unit 80 will bedescribed later in detail.

The digital camera (the electronic apparatus 1) operates in a shootingmode (multi-shot composite mode, multi-shot high resolution shootingmode) in which an image capturing operation is performed a plurality oftimes in chronological order while minutely moving the image sensor 11of the camera unit 10 in a direction different from the direction of theoptical axis of the imaging optical system (for example, within a planeorthogonal to the optical axis of the imaging optical system) using thevibration-proof unit 80. In the shooting mode (multi-shot compositemode, multi-shot high resolution shooting mode), the digital camerafurther combines these images to obtain one composite image (which isobtained not by simply adding the images but by processing image datausing special calculations), thus generating a super high-definition(high-quality, high accuracy) image. Unlike the Bayer method of therelated art that obtains only one-color information for each pixel, inthe multi-shot composite mode according to the embodiments of thepresent disclosure, color information regarding red, green, and blue(RGB) for each pixel is obtained to draw a high-definition image withmore detail and better color reproduction. Further, in the multi-shotcomposite mode according to the embodiments of the present disclosure,higher-sensitivity noise can be reduced without generating moire andfalse color.

FIGS. 2A, 2B, 2C, and 2D are diagrams for describing an example of amulti-shot composite mode according to the first embodiment. In FIGS. 2Ato 2D, the image sensor 11 includes a large number of pixels arranged ata predetermined pixel pitch in a matrix on a light-receiving surface.One of the Bayer-array color filters R, G (Gr and Gb), and B is disposedon the front surface of each pixel. Each pixel detects the color of anobject light beam that has been transmitted through the color filter R,G (Gr, Gb), or B on the front surface and that has hit the same pixel.That is, each pixel photoelectrically converts light of a colorcomponent (a particular wavelength region) into an electrical signal andobtains output according to the intensity (luminance) of the light. Morespecifically, one image is captured at the reference position of FIG.2A, and another image is captured at a position to which the light fluxregion surrounded by the thick frame has been moved downward by onepixel relative to the image sensor 11 as illustrated in FIG. 2B.Further, still another image is captured at a position (FIG. 2C) towhich the light flux region surrounded by the thick frame has beenfurther moved by one pixel from the position of FIG. 2B to the rightrelative to the image sensor 11. Then, yet another image is captured ata position (FIG. 2D) to which the light flux region surrounded by thethick frame has been further moved upward from the position of FIG. 2Cby one pixel, relative to the image sensor 11. Finally, the light fluxregion returns to the reference position in FIG. 2A. In such a manner,four images are captured in chronological order while moving (driving)the light flux region surrounded by the thick frame one pixel at a timerelative to the image sensor 11 to draw a square within the planeorthogonal to the optical axis. Then, the captured four images aretransmitted as raw image data to the image processing apparatus 20. Theimage processing apparatus 20 combines the four images captured inchronological order by the image sensor 11 to obtain a composite image.Note that in the examples of FIGS. 2A to 2D, although the light fluxregion (image capturing region) is moved by one pixel relative to theimage sensor 11, in the case of a Bayer arrangement with 4 pixels, thelight flux region (image capturing region) may be relatively moved by anodd number of pixels; the movement is not limited to one pixel. Forexample, the light flux region (image capturing region) may be movedrelative to the image sensor 11 by three pixels or by five pixels.

In the multi-shot composition using the vibration-proof unit 80, thebody of the digital camera (the electronic apparatus 1) is attached to,for example, a tripod, so as to reliably move the light flux region on apixel-by-pixel basis on the image sensor 11. In the electronic apparatus1 according to the first embodiment, the multi-shot composition isexecutable without using the vibration-proof unit 80 and also executablewith the body of the digital camera (the electronic apparatus 1) held bythe user (photographer). In other words, the electronic apparatus 1according to the embodiments of the present disclosure obtains acomposite image by the multi-shot composition based on an imagemisalignment (shift) for each shot due to camera shake (fluctuation) ofthe photographer, instead of actively moving the image sensor 11.Hereinafter, this shooting (capturing) mode is sometimes called “camerashake multi-shot composite mode”.

In the camera shake multi-shot composite mode, for example, it isdetermined whether a plurality of pieces of image data captured by thecamera unit 10 is suitable for camera shake multi-shot composition, thepixel shift amounts of the plurality of pieces of image data determinedas suitable for camera shake multi-shot composition are detected, thepieces of image data to be combined are selected from the plurality ofpieces of image data according to the detected pixel shift amounts ofthe plurality of pieces of image data, and the pieces of image data tobe combined selected according to the detected pixel shift amounts ofthe plurality of pieces of image data are relatively moved, therebyobtaining a composite image.

By operating the input device 60 of the digital camera, it is possibleto switch between the “camera shake multi-shot composite mode” and a“multi-shot composite mode using the vibration-proof unit 80”. Further,the display device 50 of the digital camera is capable of displayingwhether the “camera shake multi-shot composite mode” or the “multi-shotcomposite mode using the vibration-proof unit 80” is set. In addition tothe “camera shake multi-shot composite mode” and the “multi-shotcomposite mode using the vibration-proof unit 80” described above, theparticular shooting mode (particular image processing mode) according tothe first embodiment includes a shooting mode (an image processing mode)applied to a plurality of pieces of image data, with similarcomposition, angle, image capturing time, and image quality, that havebeen selected/extracted from a plurality of pieces of image datacontinuously shot without camera shake, or from a designated folder andcloud storage in which a set of recorded image data such as movingimages is stored. The input device 60 and the display device 50 of thedigital camera enable the particular shooting mode (particular imageprocessing mode) of a wide concept as described above to be switched anddisplayed.

The exposure changing unit 100 changes the exposure (exposure value) by,for example, adjusting the aperture value (F value) of the lens and/orthe exposure time (shutter speed). Even in the case of multi-shotcomposition in which image capturing is performed a plurality of timesat a time, the exposure can be changed for each piece of captured imagedata, and pieces of captured image data with different exposure amountscan be output. For example, the exposure changing unit 100 can changethe exposure at the time of image data acquisition at each relativemovement position of the image sensor 11 in the multi-shot composition.Furthermore, the user can also set a desired exposure (exposure value)by using the input device 60.

In multi-shot composition, images of the object are acquired, as piecesof image data, at a plurality of relative movement positions where theimaging surface of the image sensor 11 relatively moves with respect tothe object light flux. Specifically, in the multi-shot composition usingthe vibration-proof unit 80, a plurality of pieces of image data areacquired while relatively moving the imaging surface of the image sensor11 in units of pixels, with respect to the object light flux.

The exposure changing unit 100 changes the exposure (exposure value) ateach relative movement position of the image sensor 11 with respect tothe light flux of the object. For example, it is possible to increase ordecrease the exposure (exposure value) in a stepwise manner with respectto the relative movement positions of FIGS. 2A to 2D. In this case, forexample, the exposure (exposure value) may be set to an under area atthe relative movement position of FIG. 2A, the exposure (exposure value)may be set to an appropriate area at the relative movement positions ofFIGS. 2B and 2C, and the exposure (exposure value) may be set to an overarea at the relative movement position of FIG. 2D. Alternatively, theexposure (exposure value) may be set to an over area at the relativemovement position of FIG. 2A, the exposure (exposure value) may be setto an appropriate area at the relative movement positions of FIGS. 2Band 2C, and the exposure (exposure value) may be set to an under area atthe relative movement position of FIG. 2D.

FIGS. 3A to 3D illustrate an example of four pieces of image data whoseexposure is changed at each relative movement position. FIG. 3Aillustrates image data in which the exposure (exposure value) is set tothe under area, FIGS. 3B and 3C illustrate image data in which theexposure (exposure value) is set to the appropriate area, and FIG. 3Dillustrates image data in which the exposure (exposure value) is set tothe over area.

The under image has been captured in an underexposed state by reducingthe amount of light. The appropriate image has been captured with anappropriate (standard) exposure amount by which colors and tones, whichare almost as seen with the naked eye, can be reproduced. The over imagehas been captured in an overexposed state by increasing the amount oflight. The appropriate image in the present specification includes bothimage data captured with the exposure (exposure value) preferred by theuser, and image data captured with the appropriate exposure (standardexposure) corresponding to the object.

The image processing apparatus 20 has a function of adjusting thedynamic range of each piece of image data whose exposure is changed ateach relative movement position, and changing each piece of image dataaccording to the positional shift amount of each piece of image data,thereby obtaining composite image data of the pieces of image data.Hereinafter, the configuration and operational effects of the imageprocessing apparatus 20 related to this function will be described indetail. Note that in the following description, a case where four piecesof image data, acquired for multi-shot composition, are used as thepieces of image data.

FIG. 4 is a functional block diagram illustrating an example of theinternal configuration of the image processing apparatus (processor) 20.As illustrated in FIG. 4, the image processing apparatus (processor) 20includes an image data acquiring unit (image data acquiring means, imagedata input means) 21, a pixel shift amount detecting unit (pixel shiftamount detecting means, detection means) 22, a dynamic range adjustingunit (dynamic range adjusting means) 23, an image data combining unit(image data combining means) 24, and a pixel interpolating unit (pixelinterpolating means) 25.

The image data acquiring unit 21 acquires images of an object as piecesof image data at a plurality of relative movement positions where theimaging surface of the image sensor 11 relatively moves with respect tothe object light flux (acquires pieces of image data at a plurality ofrelative movement positions) (image data acquired by the image sensor 11of the camera unit 10 is input). One piece of the image data among thepieces of image data may be set as “first image data”, while anotherpiece of the image data among the pieces of image data may be set as“second image data”. The first image data and the second image data maybe output from the same image sensor 11. The first image data and thesecond image data may have different image positional relationships. Forexample, the first image data and the second image data may be capturedby different exposures from each other.

The pixel shift amount detecting unit 22 detects the pixel shift amount(positional shift amount) of each piece of image data (four pieces ofimage data) acquired by the image data acquiring unit 21. The pixelshift amount detecting unit 22 can reliably and precisely detect thepixel shift amount of each piece of image data by using a knowntechnique such as block matching disclosed in Japanese Patent No.4760923, for example.

Based on the pixel output of the image sensor 11, the pixel shift amountdetecting unit 22 can detect the pixel shift amount of each piece ofimage data, on a pixel-by-pixel basis or on a sub-pixel-by-sub-pixelbasis. Further, the pixel shift amount detecting unit 22 can detect thepixel shift amount of each piece of image data, for each RGB plane,based on the pixel output of the image sensor 11. At this time, thepixel shift amount detecting unit 22 may use only a particular RGB planeamong the RGB planes, or may change the RGB plane to be used. Forexample, it is possible to flexibly address this matter by using the Gplane when detecting the positional shift amount between certain piecesof image data, and using the R plane when detecting the positional shiftamount between other pieces of image data. Further, the pixel shiftamount detecting unit 22 may detect the positional shift amount of eachpiece of image data, based on the positional shift information of theimage data as described above and output of at least one of anacceleration detector, an angular velocity detector, and an angularacceleration detector that form the sensor 70.

The pixel shift amount detecting unit 22 can combine the detection modeusing the output of the sensor 70 described above with the detectionmode using the pixel output of the image sensor 11. That is, afterroughly estimating the directionality of the pixel shift amount by usingthe output of the sensor 70, an accurate pixel shift amount can bedetected by using the pixel output of the image sensor 11.

When pixel output of a particular application is included in the pixeloutput of the image sensor 11, the pixel shift amount detecting unit 22may detect the pixel shift amount upon interpolating or correcting thepixel output of a particular application by using other surroundingpixels, or upon excluding the pixel output of a particular application,or by applying a low weighting to the pixel output of a particularapplication. The pixel output of a particular application may include,for example, a phase difference detection pixel unrelated to imagecapturing.

The dynamic range adjusting unit 23 adjusts the dynamic range of eachpiece of image data (four pieces of image data) acquired by the imagedata acquiring unit 21. The dynamic range adjusting unit 23 adjusts thepixel output value from the image sensor 11 according to the luminanceof the object (object luminance) in each piece of image data (fourpieces of image data) acquired by the image data acquiring unit 21,thereby adjusting the level.

FIGS. 5A and 5B are conceptual diagrams illustrating an example of thedynamic range adjustment process by the dynamic range adjusting unit 23.FIG. 5A illustrates the relationship between the object luminance andthe pixel output level before the dynamic range adjustment (each pieceof image data at the time of image capturing), and FIG. 5B illustratesthe relationship between the object luminance and the pixel output levelafter the dynamic range adjustment (each piece of image data at the timeof composition). In FIGS. 5A and 5B, the pieces of image data include atotal of four images including one under image in which the exposure(exposure value) is set to the under area, one over image in which theexposure (exposure value) is set to the over area, and two appropriateimages in which the exposure (exposure value) is set to the appropriatearea.

The under image is a captured image in a range where the objectluminance is relatively high (bright). The over image is a capturedimage in a range in which the object luminance is relatively low (dark).The appropriate image is a captured image in which object luminance isin the middle range. Because the exposure amount is the same for the twoappropriate images, the object luminance is substantially overlapping.Therefore, in FIGS. 5A and 5B, the under image and the over image aredrawn by separate independent straight lines, whereas the twoappropriate images are drawn with one straight line. Specifically, thestraight line corresponding to the two appropriate images is sandwichedbetween the straight lines corresponding to under image and over image(here, the slope of the respective straight lines is the same).Incidentally, even when the object luminance is the same, the pixeloutput values are different for images with different exposure amounts.

In FIGS. 5A and 5B, overlap A is a range in which the object luminanceof the over image and the object luminance of the appropriate imageoverlap with each other, and overlap B is a range in which the objectluminance of the under image and the object luminance of the appropriateimage overlap with each other.

These four pieces of image data are combined later into one piece ofoutput image data. Therefore, in the image data after the composition,the range between the lower limit value of the object luminance in theover image and the upper limit value of the object luminance in theunder image becomes the range of the dynamic range. That is, as can beseen from the dynamic range of each of the pieces of image data at thetime of image capturing in FIG. 5A, and the dynamic range of thecomposite value of the images after the adjustment in FIG. 5B, it ispossible to obtain preferable image data in which the dynamic range isenlarged by adjusting the dynamic range by the dynamic range adjustingunit 23 based on the four pieces of image data having different exposureamounts and combining the four pieces of image data, rather thancapturing only one piece of image data.

Among the plurality of (four) captured images, the range where theobject luminance levels overlap each other, as in overlap A or overlapB, includes a plurality of pieces of pixel information. Therefore, inthe area where the object luminance levels overlap each other, theresolution of the image becomes high. With respect to the appropriateimages, a plurality of images (two images) are captured with anappropriate exposure (exposure value), and, therefore, the range of theobject luminance levels of the two appropriate images overlap with eachother (overlap C), and the resolution of the image is further increased.

That is, when focusing on the upper limit and the lower limit of theobject luminance in the appropriate image, in the area where the overlapA and the overlap C overlap each other and the object luminance isrelatively low, due to the overlapping of the object luminance levelsand the pieces of pixel information of a total of three pieces of imagedata including the two appropriate images and one over image, theresolution at the time of composition can be increased. Further, in thearea where the overlap B and the overlap C overlap each other and theobject luminance is relatively high, due to the overlapping of theobject luminance levels and the pieces of pixel information of a totalof three pieces of image data including the two appropriate images andone under image, the resolution at the time of composition can beincreased. Furthermore, even in the intermediate area of the objectluminance where the overlap C does not overlap with either overlap A oroverlap B, due to the overlapping of the object luminance levels and thepieces of pixel information of the two appropriate images, theresolution at the time of composition can be increased.

The dynamic range adjusting unit 23 can adjust the overlapping amount ofobject luminance levels in each piece of image data, by the useroperating the input device 60 or the like. The dynamic range adjustingunit 23 enlarges an area (overlap A) in which the object luminance ofthe over image and the object luminance of the appropriate imageoverlap, and/or enlarges an area in which the object luminance of theunder image and the object luminance of the appropriate image overlap(overlap B), such that it is possible to enlarge the area in which theobject luminance levels and the pieces of pixel information overlap ineach piece of image data, thereby increasing the resolution at the timeof composition.

The dynamic range adjusting unit 23 has a function of allowing the userto select a large, a medium, or a small dynamic range enlargementeffect, by operating the input device 60, etc. When the dynamic rangeenlargement effect is set to be large, the difference in the exposureamount (exposure) between the plurality of pieces of image data is setto be large when capturing images. This reduces the overlapping area ofthe object luminance levels; however, the dynamic range is furtherenlarged. Further, when the dynamic range enlargement effect is set tobe small, the difference in the exposure amount (exposure) between aplurality of images is reduced when capturing images. This increases theoverlapping area of the object luminance levels, so that the dynamicrange is enlarged and image data with higher resolution can be acquired.In this way, depending on the image capturing application of the user,it is possible to adjust the balance between the resolution and thedynamic range.

The image data combining unit 24 changes each piece of image dataaccording to the pixel shift amount (positional shift amount) of eachpiece of image data (four pieces of image data) detected by the pixelshift amount detecting unit 22, thereby obtaining composite image dataof the pieces of image data. The image data combining unit 24 obtainscomposite image data of the pieces of image data based on the pixelshift amount (positional shift amount) between the respective pieces ofimage data and the dynamic range. For example, the image data combiningunit 24 executes so-called multi-shot composition, by combining thepieces of image data acquired by the image data acquiring unit 21 toobtain composite image data. The image data combining unit 24 executes aposition matching calculation process on the pieces of image data,according to the pixel shift amount of each piece of image data detectedby the pixel shift amount detecting unit 22, and based on the result ofthis position matching calculation process, the image data combiningunit 24 relatively moves the pieces of image data, thereby obtainingcomposite image data. Here, “to relatively move each piece of imagedata” means to correct the data of each piece of image data so that thepieces of image data move relative to each other (to extract image dataupon incorporating the relative movement of each image).

FIGS. 6A and 6B are conceptual diagrams illustrating an example of animage data combining process by the image data combining unit 24. Asillustrated in FIG. 6A, as for the over image, a combining process isperformed in such a direction as to decrease the slope of the pixeloutput with respect to the object luminance, and as for the under image,a combining process is performed in such a direction as to increase theslope of the pixel output with respect to the object luminance. Further,as for the appropriate image, the image is divided into an area wherethe object luminance is relatively high and an area where the objectluminance is relatively low, and as for the area where the objectluminance is high, a combining process is performed in such a directionas to decrease the slope of the pixel output with respect to the objectluminance, and as for the area where the object luminance is low, acombining process is performed in such a direction as to increase theslope of the pixel output with respect to the object luminance. As aresult, as illustrated in FIG. 6B, composite image data is obtained bycombining one over image, one under image, and two appropriate imagesafter dynamic range adjustment. The straight line, indicating therelationship between the object luminance and the pixel output of thecomposite image data, corresponds to a line that connects a coordinateposition where the object luminance and the pixel output are minimum inthe over image, and a coordinate position where the object luminance andthe pixel output are maximum in the under image.

When combining the portion indicated by X in the composite image data ofFIG. 6B (the portion where the over image and the appropriate imageoverlap each other), there may be a defective R pixel in the under imagein some cases. In this case, for example, the composite image data maybe obtained by interpolating the defective R pixel by one of or both ofthe two appropriate images.

When combining the portion indicated by Y in the composite image data ofFIG. 6B (the portion where only the over image exists), defective pixelsmay be generated in the appropriate image and the under image in somecases. In this case, the composite image data may be obtained byacquiring, from the over image, interpolation pixels for interpolatingthe defective pixels, and interpolating the appropriate image and theunder image by using the interpolation pixels.

FIGS. 7A to 7D are conceptual diagrams illustrating an example of aprocess of combining pieces of image data after dynamic rangeadjustment. In FIGS. 7A to 7D, in each piece of image data after thedynamic range adjustment, four pixels (R′, GO′, G1′, B′) arranged in aBayer array are illustrated. Although not illustrated, in each piece ofimage data before the dynamic range adjustment, four pixels (R, GO, G1,B) arranged in a Bayer array are included. FIG. 7A corresponds to anunder image, FIGS. 7B and 7C correspond to appropriate images, and FIG.7D corresponds to an over image. The pixel shift amount between thepieces of image data is subtracted, the pixel positions in therespective pieces of image data are matched, and the positioninformation of a common target pixel is obtained. In FIG. 7A, a pixel R′is set as the target pixel, in FIG. 7B, a pixel GO′ is set as the targetpixel, in FIG. 7C, a pixel G1′ is set as the target pixel, and in FIG.7D, the pixel B′ is set as the target pixel. Then, a calculation processof matching the positions of the pieces of image data is executed, andbased on the result of this position matching calculation process, thepieces of image data are relatively moved, thereby obtaining compositeimage data. At this time, it is unnecessary to actually move/combine thepieces of image data to perform the position matching calculationprocess (only calculation is sufficient). That is, the target pixelshave an association relationship among the pieces of image data, and thetarget pixels differ in position information between the pieces of imagedata; however, the target pixels ideally indicate the same position ofthe object.

When any piece of the image data (first image data) among the pieces ofimage data includes a defective pixel (first pixel), the pixelinterpolating unit 25 interpolates the defective pixel (first pixel) inone of the pieces of image data (first image data) by using aninterpolation pixel (second pixel) of another one of the pieces of imagedata (second image data) among the pieces of image data.

As described above, when combining the portion indicated by X in thecomposite image data in FIG. 6B, a defective R pixel may be generated inthe under image in some cases. This is because in the portion X, theappropriate image and the over image are overlapping each other, but theunder image is not overlapping, and, therefore, the target pixel in theunder image becomes defective. In this case, the pixel interpolatingunit 25 can interpolate the R pixel (the target pixel that is adefective pixel), for example, by one of or both of the two appropriateimages.

FIGS. 8A to 8D are first conceptual diagrams illustrating an example ofa process of interpolating a defective pixel by the pixel interpolatingunit 25. In FIGS. 8A to 8D, in each piece of image data after thedynamic range adjustment, four pixels (R′, GO′, G1′, B′) arranged in aBayer array are illustrated. Although not illustrated, in each piece ofimage data before the dynamic range adjustment, four pixels (R, GO, G1,B) arranged in a Bayer array are included. FIG. 8A corresponds to anunder image, FIGS. 8B and 8C correspond to appropriate images, and FIG.8D corresponds to an over image. As illustrated in FIG. 8A, the R pixelthat is the target pixel in the under image, is a defective pixel Rc.

The pixel interpolating unit 25 interpolates the defective pixel Rc ofthe R pixel in the under image. The method of interpolating thedefective pixel Rc is not limited; however, as an example, by acquiringpixels from other pieces of data captured a plurality of times, it ispossible to interpolate the defective pixel with a more natural pixelvalue, rather than interpolating the defective pixel with surroundingpixels in the same image. As a matter of course, the defective pixel maybe interpolated with pixels surrounding the defective pixel in the underimage.

FIG. 9 is a second conceptual diagram illustrating an example of aprocess of interpolating a defective pixel by the pixel interpolatingunit 25. FIG. 9 is a diagram illustrating an example in which the Rpixel in the under image of FIG. 8A is the defective pixel Rc, and aninterpolation value (correction value) is calculated by referring to Rpixels surrounding the target pixel in the appropriate images in FIGS.8B and 8C, to interpolate (correct) the defective pixel Rc by using thecalculated interpolation value (correction value). As illustrated inFIG. 9, the R pixel output X at the pixel coordinate position of G2 atthe center of the appropriate image in FIG. 8B and/or the pixelcoordinate position of G7 at the center of the appropriate image in FIG.8C, can be obtained by X=(R0+R1+R2+R3)/4. Then, by multiplying the Rpixel output X by a predetermined coefficient K, the defective pixel Rcof the under image can be obtained (Rc=X·K).

Note that in the portion where the appropriate image and the under imageare overlapping but the over image is not overlapping, there is a highpossibility that a defective pixel is generated in the over image. Inthis case as well, similar to the above, it is possible to obtain aninterpolation pixel for interpolating the defective pixel in the overimage.

The portion indicated by Y in the composite image data of FIG. 6B, isformed only of the over image, and the appropriate image and the underimage are not overlapping in this portion. Therefore, in combining thepieces of image data (four pieces of image data) with the objectluminance of the Y portion, there is a possibility that the targetpixels in the two appropriate images and the one under image becomedefective pixels. In this case, the pixel interpolating unit 25interpolates the defective pixels that are the target pixels in the twoappropriate images and the one under image.

FIGS. 10A to 10D are third conceptual diagrams illustrating an exampleof a process of interpolating a defective pixel by the pixelinterpolating unit 25. As illustrated in FIG. 10A, the target pixel inthe under image is the defective pixel X′R, and as illustrated in FIGS.10B and 10C, the target pixel in each of the appropriate images is thedefective pixel X′G.

The pixel interpolating unit 25 interpolates the defective pixel X′R ofthe R pixel in the under image and the defective pixel X′G of the Gpixel in each appropriate image. The interpolation method of thedefective pixel is not limited; however, as an example, by acquiringpixels from other pieces of data captured a plurality of times, it ispossible to interpolate the defective pixel with a more natural pixelvalue, rather than interpolating the defective pixel with surroundingpixels in the same image. As a matter of course, the defective pixel maybe interpolated with pixels surrounding the defective pixel in the underimage.

FIG. 11 is a fourth conceptual diagram illustrating an example of aprocess of interpolating a defective pixel by the pixel interpolatingunit 25. As illustrated in FIG. 11, when image data of 25 pixels of 5×5is taken as an example, it is assumed that the R pixel, the G pixel, andthe B pixel positioned in the center pixel are the defective pixels XR,XG, and XB. In this case, the pixel output of the defective pixel XR isobtained by XR=R4, the pixel output of the defective pixel XG isobtained by XG=(G3+G5+G6+G7)/4, and the pixel output of the defectivepixel XB is obtained by XB=(B0+B1+B2+B3)/4. Then, the pixel outputs X′R,X′G, and X′B of the respective defective pixels used for composition areobtained, so as to match the level after the dynamic range adjustment.That is, pixel outputs X′R, X′G, and X′B corresponding to after thedynamic range adjustment are obtained based on the pixel outputs XR, XG,and XB before the dynamic range adjustment.

FIG. 12 is a first flowchart illustrating an example of an imagecapturing process according to the first embodiment.

In step ST1, it is determined whether to perform regular image capturingor multi-shot composition. When regular image capturing is to beperformed, the process proceeds to step ST2, whereas when multi-shotcomposition is to be performed, the process proceeds to step ST3.

In step ST2, one image is captured by regular image capturing, and theprocess is ended.

In step ST3, it is determined whether a dynamic range adjustment process(a dynamic range enlargement process) is to be performed. When thedynamic range adjustment process is not to be performed (step ST3: NO),the process proceeds to step ST4, and when the dynamic range adjustmentprocess is to be performed (step ST3: YES), the process proceeds to stepST5.

In step ST4, a plurality (four) of images is captured in chronologicalorder, while minutely moving the image sensor 11 by using thevibration-proof unit 80. At this time, a plurality of pieces of capturedimage data having the same exposure amount are obtained, with theexposure (exposure value) being fixed by the exposure changing unit 100.

In step ST5, a plurality of (four) images are captured in chronologicalorder while minutely moving the image sensor 11 by using thevibration-proof unit 80. At this time, the exposure (exposure value) ischanged by the exposure changing unit 100 at least once while capturingthe plurality of images, thereby obtaining a plurality of pieces ofcaptured image data having different exposure amounts.

In step ST6, a dynamic range adjustment process is executed on aplurality of pieces of image data having different exposure amountsobtained in step ST5. The dynamic range adjustment process in step ST6as a subroutine will be described in detail later.

In step ST7, an image combining process is performed with respect to theplurality of pieces of captured image data, obtained by performing thedynamic range adjustment process of step ST6 on the plurality of piecesof image data having the same exposure amount obtained in step ST4 orthe plurality of pieces of image data having different exposure amountsobtained in step ST5, thereby acquiring a single piece of image datacreated by the multi-shot composition. The process of performing theimage combining process on a plurality of pieces of image data as asubroutine will be described in detail later.

Note that the determination as to whether regular image capturing ormulti-shot composition is to be performed in step ST1 and/or thedetermination as to whether the dynamic range adjustment process is tobe performed in step ST3, may be selected or set as appropriate, forexample, by the user operating the input device 60.

FIG. 13 is a second flowchart illustrating an example of an imagecapturing process according to the first embodiment. The secondflowchart of FIG. 13 illustrates the dynamic range adjustment process ofstep ST6 of FIG. 12 as a subroutine.

In step ST61, the dynamic range adjusting unit 23 adjusts the pixeloutput level (adjustment of the dynamic range) for a plurality of piecesof image data having different exposure amounts obtained in step ST5.More specifically, the dynamic range adjusting unit 23 adjusts theoutput values of the pixels, by matching the output difference of theexposure (exposure value) at portions having the same object luminance(overlapping portions). The exposure amount of each piece of image datadiffers according to the exposure difference, and, therefore, the pixellevels are adjusted in order to combine a plurality of images havingdifferent exposure amounts.

In step ST62, the pixel interpolating unit 25 performs an interpolationprocess according to need, on a plurality of pieces of image data whosepixel output level has been adjusted (dynamic range adjustment has beenperformed). More specifically, the pixel interpolating unit 25 performspixel interpolation for a pixel that has become defected for some reason(defective pixel). The correction value may be calculated from the pixelvalues surrounding the defective pixel, or may be calculated based onpixels of other images that have been captured a plurality of times. Thedefective pixel includes, for example, a flawed pixel whose pixel valuehas not been correctly output due to the sensor, or a pixel at a portionnot overlapping at a certain object luminance. Note that theinterpolation process is not indispensable depending on the state of thecaptured image, and it can be appropriately determined whether toperform the interpolation process.

FIG. 14 is a third flowchart illustrating an example of an imagecapturing process according to the first embodiment. The third flowchartof FIG. 14 illustrates the process of performing the image combiningprocess in step ST7 of FIG. 12 as a subroutine.

In step ST71, the position matching calculation process for each pieceof image data is executed. More specifically, the image data combiningunit 24 executes the position matching calculation process for eachpiece of image data, according to the pixel shift amount of each pieceof image data detected by the pixel shift amount detecting unit 22.

In step ST72, RGB values are acquired from each piece of image data.

In step ST73, the pieces of image data are combined to acquire highresolution image data (multi-shot composition). More specifically, theimage data combining unit 24 relatively moves each piece of image databased on the result of the position matching calculation process in stepST71, thereby obtaining composite image data.

As described above, in the first embodiment, the image data acquiringmeans (10, 21) acquires image data at a plurality of relative movementpositions where the imaging surface of the image sensor 11 relativelymoves with respect to the object light flux; the exposure changing unit100 changes the exposure at the time of acquiring the image data at eachrelative movement position; the dynamic range adjusting unit 23 adjuststhe dynamic range of the image data acquired by the image data acquiringmeans (10, 21); and the image data combining unit 24 obtains compositeimage data of the pieces of image data based on the positional shiftamount between the pieces of image data and the dynamic range.Alternatively (in other words), the following processes are executed foran image set including a plurality of pieces of image data including atleast a first image and a second image whose exposure is different fromthat of the first image. That is, the pixel shift amount detecting unit22 (detecting means) detects the positional shift amount of theplurality of pieces of image data included in the image set. The dynamicrange adjusting unit 23 (dynamic range adjusting means) adjusts thedynamic range of a plurality of pieces of image data included in theimage set. The image data combining unit 24 (image data combining means)obtains composite image data of the image set based on the positionalshift amount and the dynamic range of the pieces of image data.Therefore, it is possible to realize excellent image quality with asimple configuration. As the configuration is simple, it is possible toreduce the number of images to be captured and the image capturing time.As the image quality is excellent, high resolution and a wide dynamicrange can be realized.

Conventionally, an image for increasing the resolution and an image foradjusting (enlarging) the dynamic range have been captured separately,and, therefore, the processing amount has been enormous. For example,suppose a case where four images are captured for increasing resolutionand two images are captured for widening the dynamic range. When thereis a method for each of these purposes, and the methods are combined,there is a need to capture four images for increasing the resolution twotimes, such that at least eight images need to be captured, and,therefore, an increased number of images need to be captured and moretime is taken.

On the other hand, in the first embodiment, when capturing images byshifting the pixels in order to obtain a high-resolution image, theexposure amount is changed. At this time, the images are combined uponmatching the exposure (adjusting the dynamic range) in order to obtain ahigh-resolution image. The missing parts such as highlights and shadowscan be combined with a smaller number of images than the number ofcaptured images, or a process is performed by reproducing (developing)the image with one image, thereby greatly reducing the number of imagesto be captured.

In the first embodiment described above, the case of using one overimage, one under image, and two appropriate images, as a plurality ofpieces of image data having different exposure (exposure value), hasbeen described; however, the number of pieces of image data may befreely set, and the method of setting different exposures (exposurevalues) may be freely selected, and various design changes are possible.For example, it is also possible to use two, three, or five or morepieces of image data having different exposures (exposure values).However, it is required that the object luminance overlaps between atleast two pieces of image data having different exposures (exposurevalues).

The image data output from the image data acquiring means (the cameraunit 10 and/or the image data acquiring unit 21 of the image processingapparatus 20) may be subjected to the dynamic range adjustment process(dynamic range enlargement process), after removing signal components inthe optical black area (0B) of the image data or after makingcorrections such as white balance correction in the image data.

In the image data acquiring means (the camera unit 10 and/or the imagedata acquiring unit 21 of the image processing apparatus 20), theimaging surface of the image sensor 11 is to be relatively moved withrespect to the object light flux, and there is a case of intentionallymoving the image sensor 11 by sensor driving (by the vibration-proofunit 80), and a case where the image sensor 11 passively moves(unintentionally) due to camera shake (an inadvertent movement of thecamera), etc.

Note that in the first embodiment, dynamic range adjustment of aplurality of images performed by the dynamic range adjusting unit 23(dynamic range adjusting means), may be performed any by method otherthan the above that can perform a general process of adjusting thecontrast, such as a method of combining images having differentexposures referred to as high-dynamic-range composition (HDRcomposition). Here, the high-dynamic-range composition (HDR composition)is one type of a photographing method for expressing a wider dynamicrange as compared to a regular (general) image capturing method.

Further, the order of the flow for executing the selection of thecomposition target images and the adjustment of the dynamic range, isnot particularly limited; however, it is preferable to select thecomposition target images first. This is because, after adjusting thedynamic range between a plurality of images, if these images cannot beselected as composition target images, the process has to be performedagain from the beginning.

Second Embodiment

A second embodiment will be described with reference to FIGS. 15 to 17.Descriptions of contents overlapping with the first embodiment will beomitted.

As illustrated in FIG. 15, the image processing apparatus (processor) 20includes a dividing unit 26, in addition to the image data acquiringunit 21, the pixel shift amount detecting unit 22, the dynamic rangeadjusting unit 23, the image data combining unit 24, and the pixelinterpolating unit 25.

The dividing unit 26 divides each of a plurality of images into imageareas corresponding to each other between the plurality of images. FIGS.16A to 16D are diagrams illustrating an example of dividing each of aplurality of images into image areas corresponding to each other betweenthe plurality of images. In FIG. 16A, a first image is divided intoimage areas 1-1, 1-2, . . . , 1-N in a matrix having the same size inthe vertical direction and the horizontal direction. In FIG. 16B, asecond image is divided into image areas 2-1, 2-2, . . . , 2-N in amatrix having the same size in the vertical direction and the horizontaldirection. In FIG. 16C, a third image is divided into image areas 3-1,3-2, . . . , 3-N in a matrix having the same size in the verticaldirection and the horizontal direction. In FIG. 16D, a fourth image isdivided into image areas 4-1, 4-2, . . . , 4-N in a matrix having thesame size in the vertical direction and the horizontal direction. Theblock size of each image area may be freely set; for example, the blocksize of each image area may be set as 128 pixels×128 pixels.

The pixel shift amount detecting unit 22 detects a positional shiftamount (pixel shift amount) of each of the image areas corresponding toeach other between the plurality of images. To describe this based onthe example of FIGS. 16A to 16D, the pixel shift amount detecting unit22 calculates the positional shift amount (pixel shift amount) betweenthe image area 1-1 in the first image, the image area 2-1 in the secondimage, the image area 3-1 in the third image, and the image area 4-1 inthe fourth image. Furthermore, the pixel shift amount detecting unit 22calculates the positional shift amount (pixel shift amount) between theimage area 1-2 in the first image, the image area 2-2 in the secondimage, the image area 3-2 in the third image, and the image area 4-2 inthe fourth image. Furthermore, the pixel shift amount detecting unit 22calculates the positional shift amount (pixel shift amount) between theimage area 1-N in the first image, the image area 2-N in the secondimage, the image area 3-N in the third image, and the image area 4-N inthe fourth image. In this manner, the pixel shift amount detecting unit22 calculates the correlation between blocks at the same position ineach image, for example, by subpixel estimation.

The pixel shift amount detecting unit 22 selects composite target imageareas from a plurality of images, according to the positional shiftamount (pixel shift amount) that is a correlation value detected by thepixel shift amount detecting unit 22. For example, the pixel shiftamount detecting unit 22 sets image areas in one of the images asreference image areas, sets image areas of other images as comparativeimage areas, and selects, as the composite target image area, acomparative image area based on the positional shift amount (pixel shiftamount) between the reference image area and each comparative imagearea. Specifically, the pixel shift amount detecting unit 22 selects acomparative image area whose positional shift amount (pixel shiftamount) is less than or equal to a predetermined threshold, whosepositional shift amount is smallest among the positional shift amountsbetween the reference image area and the comparative image areas, andwhose positional shift amount corresponds to an odd number of pixels oran even number of pixels. For example, when the image areas 1-1 to 1-Nof the first image in FIG. 6A are set as the reference image areas, thepixel shift amount detecting unit 22 can select at least one of theimage areas 2-1, 3-1, and 4-1 as a composite target image areacorresponding to the reference image area 1-1. Further, the pixel shiftamount detecting unit 22 can select at least one of the image areas 2-2,3-2, and 4-2 as a composite target image area corresponding to thereference image area 1-2. Still further, the pixel shift amountdetecting unit 22 can select at least one of the image areas 2-N, 3-N,and 4-N as a composite target image area corresponding to the referenceimage area 1-N.

The image data combining unit 24 obtains a composite image based on thepositional shift amount (pixel shift amount), which is the correlationvalue detected by the pixel shift amount detecting unit 22, and thecomposite target image areas selected by the pixel shift amountdetecting unit 22. The image data combining unit 24 obtains a compositeimage by executing image calculation on the composite target image areasselected by the pixel shift amount detecting unit 22, according to thepositional shift amount (pixel shift amount) that is the correlationvalue detected by the pixel shift amount detecting unit 22. For example,the image data combining unit 24 combines or replaces the referenceimage area 1-1 in FIG. 16A with the composite target image area selectedby the pixel shift amount detecting unit 22 from among the comparativeimage areas 2-1 to 4-1 in FIGS. 16B to 16D. Further, the image datacombining unit 24 combines or replaces the reference image area 1-2 inFIG. 16A with the composite target image area selected by the pixelshift amount detecting unit 22 from among the comparative image areas2-2 to 4-2 in FIGS. 16B to 16D. Further, the image data combining unit24 combines or replaces the reference image area 1-N in FIG. 16A withthe composite target image area selected by the pixel shift amountdetecting unit 22 from among the comparative image areas 2-N to 4-N inFIGS. 16B to 16D.

As described above, the image data combining unit 24 executes imagecalculation (composition or replacement) by the composite target imageareas obtained by the pixel shift amount detecting unit 22, with respectto the plurality of image areas divided by the dividing unit 26, therebyobtaining one composite image.

That is, each reference image area of one reference image is combined orreplaced with a composite target image area selected from comparativeimage areas of the comparative images. For example, the reference imagearea 1-1 of the first image (the reference image) is combined orreplaced with the composite target image area 2-1 of the second image,and the reference image area 1-2 of the first image is combined orreplaced with the composite target image area 3-2 of the third image.Further, the reference image area 1-N of the first image is combined orreplaced with a composite target image area 4-N of the fourth image.

Note that when the pixel shift amount detecting unit 22 fails to selectan appropriate composite target image area from the comparative imageareas of the comparative images for a certain reference image area ofthe reference image, the reference image area may be used as is withoutthe composition or replacement of the reference image area.

FIG. 17 is a flowchart of an image capturing process according to thesecond embodiment.

In step ST110, the dividing unit 26 divides each of a plurality ofimages into image areas corresponding to each other between theplurality of images.

In step ST120, the pixel shift amount detecting unit 22 detects thepositional shift amount (pixel shift amount) of each of the image areascorresponding to each other between the plurality of images.

In step S130, the pixel shift amount detecting unit 22 selects acomposite target image area from the plurality of images, according tothe positional shift amount (pixel shift amount) that is a correlationvalue.

In step ST140, it is determined whether a composite target image areahas been selected with respect to all image areas. When a compositetarget image area has not been selected with respect to all image areas(step ST140: NO), the process returns to step ST130 to repeat the loopof step ST130 and step ST140 until a composite target image area isselected for all the image areas. When a composite target image area hasbeen selected with respect to all image areas (step ST140: YES), theprocess proceeds to step ST150.

In step ST150, the image data combining unit 24 obtains a compositeimage based on the positional shift amounts (pixel shift amounts) thatare correlation values and the selected composite target image areas.

In the second embodiment described above, each of a plurality of imagesis divided into image areas corresponding to each other between theplurality of images, and the positional shift amount of each of theimage areas corresponding to each other between the plurality of imagesis detected. Then, a composite target image area is selected from theplurality of images based on the positional shift amount, and acomposite image is obtained based on the positional shift amounts andthe composite target image areas. Therefore, with the configurationaccording to the second embodiment of the present disclosure, ahigher-quality image (having high detail and less moiré, less falsecolor, and less high sensitivity noise, etc.) can be provided ascompared to the configuration according to the first embodiment in whichthe positional shift amount is detected on an image-by-image basis andcomposite target images are selected to obtain a composite image.

In the first embodiment, as one of the features, the exposure at thetime of acquiring the image data at each relative movement position ischanged, the dynamic range of the acquired image data is adjusted, andbased on the amount of positional shift between the respective pieces ofimage data and the dynamic range, composite image data of the pieces ofimage data is obtained.

Here, a pre-stage process of acquiring (inputting) a plurality of piecesof image data having different exposures, a middle-stage process ofadjusting the dynamic range before image composition, and a post-stageprocess of combining the pieces of image data after dynamic rangeadjustment, are assumed. For example, it is conceivable to detect thepositional shift amount of a plurality of images, select a compositetarget image from a plurality of images according to the positionalshift amount, and to obtain a composite image based on the positionalshift amounts and the composite target images. That is, it isconceivable that the pixel shifts amount of a plurality of images aredetected, one of the plurality of images is set as the reference image,and the remaining images are set as the comparative images, andaccording to the pixel shift amount between the reference image and eachcomparative image, a composite target image is selected from among thecomparative images, and the reference image and the composite targetimage are relatively moved according to the positional shift amount toobtain a composite image.

In a case where a plurality of images for adjusting the exposure/dynamicrange according to the present application, is applied to the referenceimage and the comparative image (composite target image), and theplurality of images having the adjusted exposure/dynamic range iscombined, it is possible to obtain a high-resolution (high-quality)composite image.

In the second embodiment, the case where each image is divided into aplurality of image areas, and the respective image areas are paired andthe comparison calculation is performed on the paired image areas, hasbeen described as an example. However, for example, in the case ofperforming multi-shot composition in which the image sensor isrelatively moved by one pixel at a time, it is possible to pair theimages together and perform the comparison calculation on the pairedimages, without dividing the image into image areas, pairing the imageareas, and performing the comparison calculation on the paired imageareas.

Third Embodiment

The digital camera according to the first and second embodiments doesnot drive (for example, image blur (vibration) correction drive) amoving member (for example, the image sensor 11) using thevibration-proof unit 80 in the multi-shot composite mode. However, whenthe image blur correction drive is performed crudely while using thevibration-proof unit 80 without perfectly correcting the positionalshift of a plurality of images (images are not perfectly aligned at aspecific position), the image blur correction drive is executed usingthe vibration-proof unit 80.

That is, executing the image blur correction drive using thevibration-proof unit 80 still fails to completely eliminate image blur(the image is misaligned (shifted) on the order of several microns).Accordingly, in the configuration according to the third embodiment,such an image shift (misalignment) is used in the multi-shotcomposition. This configuration is based on the concept that the amountof drive in the image blur correction drive using the vibration-proofunit 80 is significantly larger than the positional shift amount (pixelshift amount) of each image used in the multi-shot composition.

In the third embodiment, a plurality of images is obtained by, forexample, continuous shooting after setting the multi-shot composite mode(the multi-shot composite mode using camera shake, with image blurcorrection drive using the vibration-proof unit 80). Then, one compositeimage is obtained by image combining processing based on the pluralityof images.

For example, the configuration according to the third embodiment candetect the pixel shift amounts of a plurality of images, set any one ofthe plurality of images as a reference image, and set the remainingimages as comparative images. Further, the configuration can select acomposite target image from the comparative images based on the pixelshift amount between the reference image and each of the comparativeimages, and move the composite target image relative to the referenceimage based on the positional shift amount (pixel shift amount) toobtain a composite image.

Alternatively, the configuration according to the third embodiment candivide each of a plurality of images into image areas corresponding toeach other, and detect a positional shift amount of each of the imageareas corresponding to each other between the plurality of images.Further, the configuration according to the third embodiment can selecta composite target image area from the plurality of images based on thepositional shift amounts.

The configuration of the vibration-proof unit 80 is described in detailwith reference to FIGS. 18A, 18B, 19, 20, and 21. In each figure, afirst direction (Z direction and Z-axis direction) is parallel to theoptical axis O of the imaging optical system and a second direction (Xdirection and X-axis direction) is orthogonal to the first direction.Further, a third direction (Y direction and Y-axis direction) isorthogonal to both the first direction and the second direction. Forexample, assuming that the X axis, the Y axis, and the Z axis arecoordinate axes in a three-dimensional orthogonal coordinate system,when the optical axis O is designated as the Z axis, the X axis and theY axis are orthogonal to each other and both are orthogonal to theX-axis. When the digital camera is disposed in the normal position(horizontal position), the first direction (the Z direction, the Z axis,the optical axis O) and the second direction (the X direction and the Xaxis) are along the horizontal direction of the digital camera, and thethird direction (the Y direction and the Y-axis) are along the verticaldirection of the digital camera.

The digital camera (the electronic apparatus 1) includes, as a unit fordetecting vibration (fluctuation) of a camera body CB, a roll (tilt(rotation) around the Z-axis) detecting unit, a pitch (tilt (rotation)around the X-axis) detecting unit, a yaw (tilt (rotation) around theY-axis) detecting unit, an X-direction acceleration detecting unit, aY-direction acceleration detecting unit, and a Z-direction accelerationdetecting unit. Each detection unit includes a 6-axis sensor or a setconsisting of a 3-axis gyro sensor and a 3-axis acceleration sensor. Insome embodiments, each detecting unit may constitute the sensor 70 inFIG. 1.

An imaging block (for example, the camera unit 10 in FIG. 1) includes animage sensor 110A and a stage device 120 that supports the image sensor110A. The stage device 120 includes a movable stage 121 on which theimage sensor 110A is mounted, a front stationary yoke 122 on the frontof the movable stage 121, and a rear stationary yoke 123 on the back ofthe movable stage 121. The stage device 120 is capable of moving up themovable stage 121 (moved up against gravity and kept at rest) relativeto the front and rear stationary yokes 122 and 123 at least when madeconductive. The stage device 120 is capable of moving the movable stage121 in a floating state (moved up) along the Z direction (firstdirection) (parallel movement in the Z direction), along the X direction(second direction) (parallel movement in the X direction) orthogonal tothe Z direction, and along the Y direction (third direction) (parallelmovement in the Y direction) orthogonal to both the Z direction and theX direction. Further, the stage device 120 is capable of causing themovable stage 121 in a floating state (moved up) to tilt (rotate) aroundthe X-axis (second direction), around the Y-axis (third direction), andaround the Z-axis (first direction). That is, the movable stage 121 ismovable with six degrees of freedom, with respect to 6 axes.

The body CPU (for example, the CPU 90 in FIG. 1) calculates thedirection of blur and the blur speed of the digital camera based onpitch (tilting (rotation) in the X direction), yaw (tilting (rotation)in the Y direction), roll (tilting (rotation) in the Z direction), theX-direction acceleration, the Y-direction acceleration, and theZ-direction acceleration. The body CPU calculates, for example, thedrive direction, the drive speed, the drive amount of drive of the imagesensor 110A to prevent an image projected onto the image sensor 110Afrom moving relative to the image sensor 110A. Based on the calculationresults, the CPU causes the stage device 120 to travel in parallel,tilt, travel in parallel while tilting, travel in parallel aftertilting, and tilt after traveling in parallel.

The stage device 120 holds the movable stage 121, to which the imagesensor 110A is fixed, such that the movable stage 121 freely travels inparallel, tilts, travels in parallel while tilting, and travels inparallel after tilting relative to the front stationary yoke 122 and therear stationary yoke 123. The movable stage 121 is a rectangular platemember larger than the image sensor 110A when viewed from the front. Thefront stationary yoke 122 and the rear stationary yoke 123 arerectangular frame members each having the same shape and an outer shapelarger than the movable stage 121 in plan view. Each of the frontstationary yoke 122 and the rear stationary yoke 123 has a rectangularopening (122 a/123 a) larger than the outer shape of the image sensor110A at the central portion of each of the front stationary yoke 122 andthe rear stationary yoke 123, when viewed from the front (the Zdirection).

The front stationary yoke 122 has an X-direction magnet MX on at leastone side of the right and left (X direction) of the opening 122 a withrespect to the Z-axis with the Y-axis as the center line on the back(the surface opposite to the object side). However, in the embodiment asillustrated in FIGS. 18A and 18B, an X-direction magnet MX is disposedon each side of the right and left of the opening 122 a. That is, a pairof X-direction magnets MX, each made of a permanent magnet having thesame specification, is fixed to the back surface of the front stationaryyoke 122. By passing the magnetic flux of the X-direction magnets MXthrough the front stationary yoke 122 and the rear stationary yoke 123,a magnetic circuit that generates thrust in the X direction (the seconddirection) is formed between the X-direction magnets MX on the right andleft sides and the opposed portion of the rear stationary yoke 123.

The front stationary yoke 122 has a pair of a Y-direction magnet MYA anda Y-direction magnet MYB at the lower side relative to the opening 122 aon the back of the front stationary yoke 122. The magnet MYA and themagnet MYB are opposed to each other across the Y-axis as the centerline and away from the Z-axis. Each of the magnet MYA and the magnet MYBis a permanent magnet having the same specification. By passing themagnetic flux of the magnet MYA and the magnet MYB through the frontstationary yoke 122 and the rear stationary yoke 123, a magnetic circuitthat generates thrust in the Y direction (the third direction) is formedbetween the Y-direction magnet MWA and the Y-direction magnet MWB andthe rear stationary yoke 123.

The front stationary yoke 122 also has Z-direction magnets MZA, MZB, andMZC fixed onto three positions away from the Y-direction magnets MYA andMYB on the back surface. The magnets MZA, MZB, and MZC are permanentmagnets of the same specification. The three Z-direction magnets MZA,MZB, and MZC are disposed at substantially equal intervals in a planeorthogonal to the Z-axis with the Z axis as the center of the plane. Bypassing a magnetic flux of the Z-direction magnets MZA, MZB, and MZCthrough the front stationary yoke 122 and the rear stationary yoke 123,a plurality of magnetic circuits that generates thrust in the Zdirection (the first direction) is formed between the Z-directionmagnets MZA, MZB, and MZC and the rear stationary yoke 123.

The movable stage 121 has a hole 121 a for the image sensor 110A at thecenter portion of the movable stage 121. The hole 121 a is rectangularwhen viewed from the front. The image sensor 110A is fit in the hole 121a. The image sensor 110A projects forward beyond the hole 121 a in thedirection of the optical axis O of the movable stage 121.

The movable stage 121 further has a pair of X-drive coils CX and a pairof a Y-drive coil CYA and a Y-drive coil CYB. The X-drive coils CX arefixed onto the outer portions of the right and left sides (short sides)of the image sensor 110A, respectively. The Y-drive coil CYA and theY-drive coil CYB are fixed onto the lower portion of the image sensor110A (in the vicinity of the lower side (long side) of the image sensor110A), apart from each other along the right-to-left direction of theimage sensor 110A. The movable stage 121 further has a circular Z-drivecoil CZA and a pair of circular Z-drive coils CZB and CZC. The Z-drivecoil CZA is stationary (in the intermediate position) between theY-drive coils CYA and CYB. The Z-drive coils CZB and CZC are stationaryat the upper position relative to the pair of the X-drive coils CX.

The above-described X-drive coil CX, the Y-drive coil CYA, the Y-drivecoil CYB, the Z-drive coil CZA, the Z-drive coil CZB, and the Z-drivecoil CZC are connected to an actuator drive circuit that controls powerdistribution.

In the movable stage 121, X-direction Hall elements HX are fixed in theair core areas of the X-drive coils CX, and a Y-direction Hall elementHYA and a Y-direction Hall element HYB are fixed in the air core areasof the Y-drive coils CYA and CYB, respectively. Further, Z-directionHall elements HZA, HZB, and HZC are fixed in the air core areas ofZ-drive coils CZA, CZB, and CZC, respectively.

A position detection circuit detects the position of the movable stage121 in the X direction, the position in the Y direction, the position inthe Z direction, the position of tilt rotation around the X-axis (tiltrotation angle around the X-axis and pitch angle), the position of tiltrotation around the Y-axis (tilt rotation angle around the Y-axis andYaw angle), and the position of tilt rotation around the Z-axis (tiltrotation angle around the Z-axis and roll angle), based on detectionsignals output from X-direction Hall elements HX, the Y-direction Hallelements HYA and HYB, and Z-direction Hall elements HZA, HZB, and HZC.

Based on the detection result of the position detection circuit, theactuator drive circuit drives the image sensor 110A (the movable stage121) by controlling power distribution to the X-drive coils CX, theY-drive coils CYA, CYB, CZA and the Z-drive drive coils CZA, CZB, andCZC. For example, the vibration-proof unit 80 serves as a camera shakecorrection device (drive device) that corrects image blur (vibration) bydriving (moving) the image sensor 110A, which is a part of theimage-capturing device, as a drive member in a direction different fromthe direction of the optical axis O (Z-axis) of the image-capturingdevice. Note that the drive member to be driven is not limited to theimage sensor 110A, and may be, for example, an image-blur correctionlens as a part of the photographing lens.

The present inventor has conceived of the following concept throughintensive studies of a technique of executing multi-shot composite whileexecuting image-blur correction drive using the above-described hexaxialdrive unit (however, the mode of image-blur correction is not limited)as one example. Even if a parallel-direction shift of the drive member(image sensor) remains within a plane (XY plane) orthogonal to theoptical axis O (Z axis), the image quality of the multi-shot compositeis not adversely affected. However, it is found that if arotational-direction shift of the drive member (image sensor) remainswithin the plane (XY plane) orthogonal to the optical axis O (Z axis),the image quality of the multi-shot composite is adversely affected.

As described above, in the embodiments of the present disclosure, theimage calculation such as detection of the positional shift amount(pixel shift amount) of a plurality of images or image areas isperformed based on the XY coordinate axes in the XY plane. Accordingly,when a rotational shift within the XY plane is large, correlationbetween a plurality of images or between a plurality of image areascannot be obtained, and appropriate image calculation may be difficult.

FIGS. 22A and 22B are diagrams of adverse effects of image blur (shift,vibration) in the rotational direction within the XY plane. Asillustrated in FIGS. 22A and 22B, the image blur amount in therotational direction within the XY plane decreases with a reduction indistance to the optical axis O (Z-axis) (closer to the center of theimage), and increases with an increase in distance to the optical axis O(Z-axis) (closer to the periphery of the image).

In the embodiments of the present disclosure, not only the shift amountin the parallel direction within a plane (the XY plane) orthogonal tothe optical axis O (Z axis) but also the shift amount in the rotationaldirection within the plane orthogonal to the optical axis O (Z axis)(the XY plane) is corrected using the vibration-proof unit 80. With sucha configuration, the accuracy of the image calculation can be increasedand the image quality of the multi-shot composite can be improved aswell. Further, the processing load and the processing time of the imagecalculation can be reduced.

In some embodiments, the vibration-proof unit (drive device) 80 mayrelatively reduce the drive component (drive amount) of the drive member(image sensor) in the parallel direction within a plane (XY plane)orthogonal to the optical axis O (Z axis), and relatively increase thedrive component (drive amount) of the drive member (image sensor) in therotational direction within the plane (XY plane) orthogonal to theoptical axis O (Z axis). This configuration permits a certain amount ofparallel-direction shift components (shift amount) of the driving member(image sensor) to remain in the XY image, which has a small adverseeffect on the image quality of the multi-shot composite. Further, such aconfiguration positively eliminates the rotational-direction shiftcomponents (shift amount) of the driving member (image sensor) toprevent a significant adverse effect on the image quality to increasethe image quality of the multi-shot composite.

Further, as in the second embodiment, by dividing each of a plurality ofimages into image areas corresponding to each other by the dividing unit26 and calculating a positional shift amount (pixel shift amount) foreach image area, the shift amount of drive member (image sensor) in therotational direction can be reduced.

In this case, the dividing unit 26 preferably divides each of aplurality of images into image areas having different sizes. Morespecifically, the dividing unit 26 preferably divides the centerportions of each of the plurality of images preferably into image areasof relatively large sizes, and divides the peripheral portions of eachof the plurality of images into image areas of relatively small sizes.

FIG. 23 is a diagram of an example in which a plurality of images isdivided into image areas having different sizes. In FIG. 23, the imagearea is formed by a total of 80 blocks in the minimum block unit, thatis, eight blocks in the vertical direction x ten blocks in thehorizontal direction. The image area in FIG. 23 is divided into amaximum image area block in the center portion of the image area, twointermediate image area blocks on each side of the maximum image area,and minimum image area blocks on the periphery of the image area,surrounding the maximum image area block and the intermediate image areablocks. The maximum image area block has a size of 16 (four-by-fourpixels) minimum image area blocks (minimum block unit). The intermediateimage area block has a size of 4 (two-by-two pixels) minimum image areablocks (minimum block unit).

For example, when there is a shift in the rotation direction among aplurality of images, the shift amount decreases toward the centerportion of the image, and increases toward the periphery of the image(see FIGS. 22A and 22B). In view of this, the image area correspondingto the center portion of the image in which the shift amount in therotational direction is small is divided into large (coarse) blocks,while the image area in the periphery of the image in which the shiftamount in the rotational direction is large is divided into small (fine)blocks. Accordingly, the accuracy of image calculation in each imagearea block (particularly in the image area blocks in the periphery ofthe image) can be increased, and image quality of the multi-shotcomposite can be improved. Further, the processing load and theprocessing time of the image calculation can be reduced. In FIG. 23, ifall the image area blocks are divided into the minimum image area blocks(minimum block units), the processing load of the image calculation andthe processing time increase. Further, in FIG. 23, if all the image areablocks are divided into the maximum image area blocks, correlationbetween the image area blocks might not be obtained (the pixel shiftamount might not be calculated) in the image peripheral portion in whichthe shift amount in the rotation direction is large.

According to one embodiment of the present invention, an image capturingapparatus, an image capturing method, and an image processing apparatusthat are simple in structure and that can achieve excellent imagequality, are provided.

The image capturing apparatus, the image capturing method, and the imageprocessing apparatus are not limited to the specific embodimentsdescribed in the detailed description, and variations and modificationsmay be made without departing from the spirit and scope of the presentinvention.

What is claimed is:
 1. An image capturing apparatus comprising: an imagedata acquirer configured to acquire pieces of image data at a pluralityof relative movement positions to which an imaging surface of an imagingsensor is relatively moved with respect to an object light flux; anexposure changer configured to change an exposure when acquiring thepieces of image data at the plurality of relative movement positions; adynamic range adjuster configured to adjust a dynamic range of theacquired pieces of image data; and an image data combiner configured toobtain composite image data of the pieces of image data, based on apositional shift amount between the pieces of image data and the dynamicrange.
 2. The image capturing apparatus according to claim 1, whereinthe dynamic range adjuster adjusts an overlapping amount of objectluminance levels in the pieces of image data.
 3. The image capturingapparatus according to claim 1, further comprising: a pixel interpolatorconfigured to interpolate a first pixel in first image data among thepieces of image data, by a second pixel in second image data among thepieces of image data, when the first pixel is included in the firstimage data.
 4. The image capturing apparatus according to claim 3,wherein the first pixel is a defective pixel and the second pixel is aninterpolation pixel.
 5. An image capturing method comprising: acquiringpieces of image data at a plurality of relative movement positions towhich an imaging surface of an imaging sensor is relatively moved withrespect to an object light flux; changing an exposure when acquiring thepieces of image data at the plurality of relative movement positions;adjusting a dynamic range of the acquired pieces of image data; andobtaining composite image data of the pieces of image data, based on apositional shift amount between the pieces of image data and the dynamicrange.
 6. An image processing apparatus comprising: a detectorconfigured to detect a positional shift amount of a plurality of piecesof image data included in an image set, the image set including theplurality of pieces of image data of at least a first image and a secondimage with an exposure different from an exposure of the first image; adynamic range adjuster configured to adjust a dynamic range of theplurality of pieces of image data included in the image set; and animage data combiner configured to obtain composite image data of theimage set, based on the positional shift amount of the plurality ofpieces of image data and the dynamic range.